{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# 中文文档转换实战\n",
    "- 文档切割器和按字符分隔\n",
    "- 代码文档分割器\n",
    "- 按Token分割文档\n",
    "- 文档总结、精炼、翻译\n"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "82c5105a74c793e9"
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 文档切割\n",
    "### 原理\n",
    "1. 将文档分成小的、有意义的块(句子)\n",
    "2. 将小的块组成更大的块，直到达到一定的大小\n",
    "3. 一旦达到一定的大小，就接着创建与下一个块重叠的部分\n",
    "\n",
    "\n",
    "### 示例\n",
    "- 第一个文档分割\n",
    "- 按字符切割\n",
    "- 代码文档切割\n",
    "- 按token切割\n"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "498b5a24cd251125"
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### 第一个文档分割\n",
    "将按不同的字符递归地分割(按照这个优先级[\"\\n\\n\", \"\\n\", \" \", \"\"]),"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "74e5b78bfac3a05f"
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "page_content='蒂法介绍' metadata={'start_index': 1}\n"
     ]
    },
    {
     "data": {
      "text/plain": "Document(page_content='蒂法·洛克哈特(日语:ティファ?ロックハート，Tifa Rokkuhāto，英语:Tifa', metadata={'start_index': 6})"
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
    "\n",
    "# 加載要切分的文档\n",
    "with open(\"test.txt\") as f:\n",
    "    content = f.read()\n",
    "\n",
    "# 使用递归字符切分器\n",
    "splitter = RecursiveCharacterTextSplitter(\n",
    "    chunk_size=50,#切分的文本块大小，一般通过长度函数计算\n",
    "    chunk_overlap=20,#切分的文本块重叠大小，一般通过长度函数计算\n",
    "    length_function=len,#长度函数,也可以传递tokenize函数\n",
    "    add_start_index=True,#是否添加开始索引\n",
    ")\n",
    "text = splitter.create_documents([content])\n",
    "print(text[0])\n",
    "text[1]"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-15T13:51:00.410953700Z",
     "start_time": "2024-05-15T13:51:00.404737Z"
    }
   },
   "id": "fe5d83774fa9f882"
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### 按字符切割"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "d500bd38a7240bb1"
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Created a chunk of size 125, which is longer than the specified 50\n",
      "Created a chunk of size 72, which is longer than the specified 50\n",
      "Created a chunk of size 72, which is longer than the specified 50\n",
      "Created a chunk of size 63, which is longer than the specified 50\n",
      "Created a chunk of size 52, which is longer than the specified 50\n",
      "Created a chunk of size 96, which is longer than the specified 50\n",
      "Created a chunk of size 51, which is longer than the specified 50\n",
      "Created a chunk of size 66, which is longer than the specified 50\n",
      "Created a chunk of size 105, which is longer than the specified 50\n",
      "Created a chunk of size 84, which is longer than the specified 50\n",
      "Created a chunk of size 78, which is longer than the specified 50\n",
      "Created a chunk of size 72, which is longer than the specified 50\n",
      "Created a chunk of size 66, which is longer than the specified 50\n",
      "Created a chunk of size 92, which is longer than the specified 50\n",
      "Created a chunk of size 58, which is longer than the specified 50\n",
      "Created a chunk of size 67, which is longer than the specified 50\n",
      "Created a chunk of size 73, which is longer than the specified 50\n",
      "Created a chunk of size 52, which is longer than the specified 50\n",
      "Created a chunk of size 61, which is longer than the specified 50\n",
      "Created a chunk of size 77, which is longer than the specified 50\n",
      "Created a chunk of size 82, which is longer than the specified 50\n",
      "Created a chunk of size 61, which is longer than the specified 50\n",
      "Created a chunk of size 52, which is longer than the specified 50\n",
      "Created a chunk of size 60, which is longer than the specified 50\n",
      "Created a chunk of size 51, which is longer than the specified 50\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "page_content='蒂法介绍\\n蒂法·洛克哈特(日语:ティファ?ロックハート，Tifa Rokkuhāto，英语:Tifa Lockhart)为电子游戏《最终幻想VII》及《最终幻想VII补完计划》相关作品中的虚构? 色，由?村哲也创作和设计，此后也在多个游戏中客串登场' metadata={'start_index': 1}\n"
     ]
    }
   ],
   "source": [
    "from langchain.text_splitter import CharacterTextSplitter\n",
    "# 加载要切分的文档\n",
    "with open(\"test.txt\") as f:\n",
    "    content = f.read()\n",
    "splitter = CharacterTextSplitter(\n",
    "    separator=\"。\", #切割的标志字符，默认的是\\n\\n\n",
    "    chunk_size=50,#切分的文本块大小，一般通过长度函数计算\n",
    "    chunk_overlap=20,#切分的文本块重叠大小，一般通过长度函数计算\n",
    "    length_function=len,#长度函数,也可以传递tokenize函数\n",
    "    add_start_index=True,#是否添加开始索引\n",
    ")\n",
    "text = splitter.create_documents([content])\n",
    "print(text[0])"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-15T13:54:13.633031200Z",
     "start_time": "2024-05-15T13:54:13.574870700Z"
    }
   },
   "id": "797e16bfc79fb696"
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### 代码文档切割"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "41d979e6639940c8"
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cpp\n",
      "go\n",
      "java\n",
      "js\n",
      "php\n",
      "proto\n",
      "python\n",
      "rst\n",
      "ruby\n",
      "rust\n",
      "scala\n",
      "swift\n",
      "markdown\n",
      "latex\n",
      "html\n",
      "sol\n"
     ]
    }
   ],
   "source": [
    "from langchain.text_splitter import RecursiveCharacterTextSplitter, Language\n",
    "\n",
    "# 支持的语言\n",
    "for e in Language:\n",
    "    print(e.value)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-15T14:06:49.592153800Z",
     "start_time": "2024-05-15T14:06:49.584869600Z"
    }
   },
   "id": "dfd6ff74e5df20fe"
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "outputs": [
    {
     "data": {
      "text/plain": "[Document(page_content='def hello_world():\\n    print(\"Hello, World!\")', metadata={}),\n Document(page_content='#调用函数\\nhello_world()', metadata={})]"
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 加载要切分的代码段\n",
    "PYTHON_CODE = \"\"\"\n",
    "def hello_world():\n",
    "    print(\"Hello, World!\")\n",
    "#调用函数\n",
    "hello_world()\n",
    "\"\"\"\n",
    "\n",
    "py_splitter = RecursiveCharacterTextSplitter.from_language(\n",
    "    chunk_size=50,#切分的文本块大小，一般通过长度函数计算\n",
    "    chunk_overlap=20,#切分的文本块重叠大小，一般通过长度函数计算\n",
    "    language=Language.PYTHON\n",
    ")\n",
    "\n",
    "python_docs = py_splitter.create_documents([PYTHON_CODE])\n",
    "python_docs # 分割成两段文档"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-15T14:09:17.699041Z",
     "start_time": "2024-05-15T14:09:17.692210100Z"
    }
   },
   "id": "42c14b2c5d21b1d6"
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### 按token切割"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "b2ce04055d010336"
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Created a chunk of size 3110, which is longer than the specified 500\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "page_content='蒂法介绍\\n蒂法·洛克哈特(日语:ティファ?ロックハート，Tifa Rokkuhāto，英语:Tifa Lockhart)为电子游戏《最终幻想VII》及《最终幻想VII补完计划》相关作品中的虚构? 色，由?村哲也创作和设计，此后也在多个游戏中客串登场。 2014年东京电玩展上，星名美津纪cosplay《最终幻想VII 降临之子》中的蒂法·洛克哈特 蒂法是克劳德的?梅竹?，两人同为尼布鲁海姆出身。在米德加经营作为反抗组织“雪崩”根 据地的酒馆“第七天堂”，并且是小有名气的招牌店员。擅?格斗，以拳套为武器。本传7年前 克劳德离开故乡从军时，曾许下约定“如果有危机时一定会保护她”。与爱丽丝相识之后，两 人成为好友。第一个察觉克劳德记忆混乱的人，后来协助精神崩溃的克劳德?新找回真正的自 己。本传的大战结束后，依大家的期待在战后新生的米德加再次开设第七天堂(原第七天堂因 第柒区圆盘崩塌遭压毁)，同时也照顾一群受到星痕症候群折磨的孩子们。 蒂法被《纽约时报》称为“网络一代”的海报女郎，与劳拉·克罗夫特相比，她是电子游戏中坚 强、?立和有吸引力的女性?色的典型代表。媒体普遍称赞其实力和外表，并称她为游戏世界 中最好的女性?色之一。 在《最终幻想VII》本传中，蒂法年龄20岁、身高167cm、生日5月3日、血型B型、出生地尼 布尔海姆。\\n登场\\n《最终幻想VII》 蒂法在《最终幻想VII》原版中首次亮相，是克劳德的?梅竹?、第七天堂酒吧的看板娘、极 端环境组织“雪崩”成员，该组织反抗巨型企业“神罗”因其大?抽取魔晄用作动力能源。在注 意到克劳德的性格改变后，她说服克劳德加入雪崩，以密切关注他，并且跟随他追寻游戏中的 对手萨菲罗斯。她无法阻止克劳德被萨菲罗斯操纵，在他的精神崩溃后，她帮助克劳德康复， 并且两人意识到彼此间的相互感觉，最后与伙伴们一同击败了萨菲罗斯。[2] 在闪回中可知，儿时的蒂法一直是村中小孩的人气王。在母亲过世后，思念母亲的蒂法决定沿 着小路走到他们故乡尼布尔海姆附近的一座山上，认为这样就能?到过世的母亲，原本跟着蒂 法的其他小孩都在半路上因害怕而放弃，唯?克劳德仍坚定的在后面跟随，希望能在危机时保 护蒂法。然而，他们俩都从山上跌落受伤，蒂法昏迷了一个星期，她的父亲认为克劳德对此负 有责任[3]，甚为严令禁止克劳德再接近蒂法，但蒂法反而从此更在意克劳德，两人成为要好 的玩伴。为了使自己变得更强大，克劳德最终选择离开尼布尔海姆，加入神罗，想要成为神罗 的精英战士“神罗战士”(SOLDIER)，但后来透露他主要是为了吸引蒂法的注意力。离开之 前，蒂法与克劳德约定，当蒂法处于困境之中时，克劳德会回来救她。从克劳德离开之后，蒂 法便开始留意神罗战士的消息，因为神罗战士都成为声名远播的知名人物，如果克劳德成为神 罗战士，他的活跃也会立刻传回尼布尔海姆。数年后，在萨菲罗斯摧毁了尼布尔海姆之后，克 劳德为了救蒂法，被萨菲罗斯刺至?伤。蒂法被她的武术教练赞干带到安全地带，幸存下来， 最终到达米德加并遇?了“雪崩”的领导人巴雷特·华莱士。病愈后，蒂法加入了“雪崩”，为 了给家乡被毁一事报仇。一天，她在火?站遇到了从魔晄炉中逃出来、精神一片混乱的克劳 德，蒂法说服了他为巴雷特工作，以保证克劳德的安全以及和克劳德保持紧密关系。这是游戏 开始的地方。 在原版《最终幻想VII》中蒂法与爱丽丝关系友好，但会在某些时候争?吃醋，例如在神罗总 部营救爱丽丝时，蒂法及巴雷特等一行失手被擒，若克劳德选择关心爱丽丝的话蒂法的对话中 明显带有妒忌。在?制版中虽然删去这段情节，但保留了这种关系。 在《最终幻想VII》的初稿中，蒂法是背景人物。她在“雪崩”中的作用是在幕后支持，在执 行任务后为所有人加油鼓劲，并且对克劳德有特别的关心。据推测，她的背上有一块大的疤' metadata={}\n"
     ]
    }
   ],
   "source": [
    "from langchain.text_splitter import CharacterTextSplitter\n",
    "\n",
    "# 加载要切分的文档\n",
    "with open(\"test.txt\") as f:\n",
    "    content = f.read()\n",
    "    \n",
    "text_splitter = CharacterTextSplitter.from_tiktoken_encoder(\n",
    "    chunk_size=500,#切分的文本块大小，一般通过长度函数计算\n",
    "    chunk_overlap=5,#切分的文本块重叠大小，一般通过长度函数计算\n",
    ")\n",
    "text = text_splitter.create_documents([content])\n",
    "print(text[0])"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-05-15T14:11:16.018674500Z",
     "start_time": "2024-05-15T14:11:16.009442300Z"
    }
   },
   "id": "e4479eeb99a90cb9"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [],
   "metadata": {
    "collapsed": false
   },
   "id": "a72ea9538a46d85b"
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
